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ABSTRACT 


To  achieve  the  long-term  environmental  restoration  objectives  of  the  Department  of 
Energy,  the  Office  of  Technology  Development  is  developing  remediation  technologies  that  are 
better,  faster,  safer,  and  more  cost  effective  than  those  currently  available.  These  innovative 
technologies  approach  the  state-of-the-art,  with  commensurately  more  risky  development 
efforts  than  with  more  established  technology.  This  study  applies  multi-attribute  utility  theory 
to  the  DOE’S  problem  of  selecting  the  least  risky  technologies  to  develop.  It  focuses  on  two 
aspects  of  the  technical  risks  involved  with  emerging  technologies:  the  risk  of  successful 
development  (measured  by  the  likelihood  of  successful  R&D  completion  within  seven  years) 
and  the  risk  of  successful  implementation  in  the  field  (measured  by  the  likelihood  of  the 
technology  performing  as  anticipated).  Three  experienced  contractors  working  with  the 
LandfiU  Stabilization  Focus  Area  served  in  the  roles  of  DOE  technology  managers.  Their 
preferences,  expressed  as  MAUT  utility  functions,  were  assessed  for  all  seven  functional 
remediation  processes.  These  utilities  are  plotted  as  three-dimensional  surfaces  over  a  plane 
formed  by  the  two  technical  risk  measures,  graphically  depicting  the  trade-offs  each  “decision 
maker”  felt  was  appropriate  between  the  two  measures.  Their  utility  surfaces  for  treatment 
technologies  were  then  used  with  a  set  of  ten  treatment  methods  for  which  expert  estimates  of 
the  two  risk  measures  were  available.  The  resulting  rankings  of  technologies  are  analyzed  to 
examine  the  suitability  of  the  methodology.  Recommendations  for  the  use  of  this  approach 
with  real  DOE  managers  are  presented. 
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1.  Introduction 


1.1  Background 

Although  positive  steps  have  been  taken  during  the  past  thirty  years  to  remedy  the  nation’s 
environmental  problems,  many  challenges  remain.  To  continue  to  address  these  challenges,  the 
U.  S.  Department  of  Energy  (DOE)  has  been  implementing  an  aggressive  national  program  of 
applied  research  that  encourages  the  development  of  technologies  to  meet  restoration  and  waste 
management  needs,  with  highest  priority  being  given  to  DOE’s  most  pressing  major 
environmental  management  problems.  The  keystone  of  DOE’s  approach  is  to  develop 
remediation  technologies  that  are  better,  faster,  safer,  and  more  cost  effective  than  those  currently 
available  [DOEa,  1995a:vii-viii].  These  innovative  technologies  approach  the  frontier  of  the 
current  state-of-the-art.  For  this  reason,  DOE  lacks  historical  data  on  which  to  base  predictions 
of  the  proposed  technologies’  eventual  successful  development.  As  these  technologies  progress 
toward  eventual  employment,  the  DOE  will  be  driven  by  limited  budgets  to  fully  fund  only  the 
most  promising  approaches.  Estimating  the  new  technologies’  future  performance  is  of  crucial 
importance  to  DOE  funding  decisions,  despite  the  difficulty  involved. 

Risks,  to  a  program  manager,  relate  to  delivering  a  specified  product  or  level  of 
performance  at  a  specified  time  for  a  specified  cost.  A  wide  variety  of  problems  can  prevent  the 
meeting  of  these  cost,  schedule,  and  performance  objectives.  Failure  to  meet  these  goals,  and  the 
anticipated  probability  of  that  failure,  form  the  risk  in  the  program. 
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While  there  are  other  sources  of  program  risk,  including  management  difficulties,  funding 
delays,  and  other  circumstantial  effects,  a  great  deal  of  risk  can  be  associated  with  the  technology 
being  developed  itself.  The  attempt  to  provide  a  new  or  greater  level  of  performance  than 
previously  demonstrated,  or  a  similar  level  of  performance  subject  to  some  new  constraints  of 
budget,  packaging,  or  time,  carries  with  it  the  possibility  of  failure  with  the  consequence  of  lost 
time  and  money.  This  risk  is  generally  referred  to  as  “technical”  or  “technological  risk,”  and  is  of 
critical  importance  to  projects  trying  to  improve  on  the  state-of-the-art  [DSMC,  1989:3-3]. 

One  important  concern  of  program  managers  investing  in  these  innovative  remediation 
technologies  is  “Will  it  work,  at  least  as  well  as  specified?”  Since  we  cannot  truly  know  this  until 
the  technology  has  been  fielded,  we  are  forced  to  rely  on  expert  judgement  and  modeling  to  form 
estimates  of  the  eventual  level  of  performance. 

The  decision,  however,  is  not  as  simple  as  merely  choosing  the  technologies  most  likely  to 
succeed  in  the  field.  Some  technologies  may  have  excellent  potential  for  remediation  use,  but  may 
rely  on  advances  in  the  state-of-the-art  that  are  only  now  being  proposed  or  developed.  These 
technologies  may  not  be  ready  for  field  use  for  many  years,  and  so  may  not  be  the  right  choice  for 
certain  sites  despite  their  expected  usefulness. 

This  study  concerns  itself  with  the  tradeoffs  between  these  two  important  technical  risks 
—  the  risk  that  a  technology  may  not  be  successful  in  real-world  use,  and  the  risk  that  a 
technology  will  not  be  successfully  developed  in  time  for  some  expected  application.  A  manager 
facing  the  decision  of  investing  in  tomorrow’s  remediation  solutions  will  value  technologies  with 
different  estimated  risks  based  on  his  or  her  personal  preferences  formed  by  past  experience  and 
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present  circumstances.  But  even  a  veteran  DOE  manager  may  have  difficulty  choosing  between 
many  candidate  technologies  with  similar  estimated  risks. 

This  report  describes  the  use  of  multi-attribute  utility  theory  to  assist  in  such  a  decision,  by 
forming  a  mathematical  model  of  the  manager’s  preferences  and  using  that  model  to  distinguish 
between  candidate  technologies.  This  model  is  not  meant  to  replace  the  manager’s  judgement, 
but  to  clarify  the  technical  risk  aspect  of  the  technology  selection  decision.  He  or  she  must 
examine  other  issues,  such  as  funding  limitations,  timeliness,  political  concerns,  and  so  on,  in 
addition  to  each  technologies’  technical  risk  when  making  the  decision.  These  other  issues  could 
also  be  the  subject  of  similar  utility  models  as  desired. 

After  some  background  discussion  of  methodology,  models  developed  for  three  different 
“decision  makers”  will  be  described.  These  “decision  makers”  were  actually  contractors  working 
for  the  Landfill  Stabilization  Focus  Area  field  office  at  the  Savannah  River  Site  in  Aiken,  South 
Carolina.  They  were  interviewed  in  early  December  1995  to  assess  their  own  preferences  for 
various  levels  of  the  technical  risk  criteria  already  described.  The  models  of  these  collected 
preferences  will  then  be  used  on  the  technology  selection  decision  posed  in  Figure  1.1.  The  ten 
treatment  technologies  shown  on  the  diagram  were  selected  by  Dr  Paul  Krumrine  of  the  Waste 
Policy  Institute  for  this  sample  decision  problem  because  estimates  of  the  probabilities  of 
successful  development  and  use  were  easily  gathered.  Four  different  estimates  of  these  attributes 
were  used  to  rank  order  the  candidate  technologies  with  respect  to  the  three  different  decision 
makers’  preferences.  The  results  of  these  twelve  combinations  of  estimates  and  preference 
models  will  then  be  discussed. 
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1.2  Methodology 

Figure  1.2  describes  the  overall  approach  to  the  technology  selection  problem  in 
Figure  1.1.  The  heart  of  the  matter  is  diagrammed  in  Figure  1.3.  Given  a  time  horizon  and  a  list 
of  technologies  to  examine,  expert  opinion  is  used  to  develop  estimates  of  the  two  technical  risk 
factors  for  each  technology.  Then  the  preferences  of  our  decision  maker  are  modeled 
mathematically,  resulting  in  numerical  expressions  of  the  technical  risk  value  of  these  technologies 
with  respect  to  these  technical  risks.  The  overall  score  incorporates  the  trade-offs  between  the 
two  risk  factors  that  are  acceptable  to  the  decision  maker.  The  technical  risk  factors  are  also 
called  “attributes”  in  the  context  of  multi-attribute  utility  theory. 

1.2.1  Attributes. 

1.2. 1.1  The  risk  of  successful  development,  P(dev).  There  is  no  guarantee  that  an 
ambitious  technological  approach  will  be  successful  —  one  estimate  of  the  likelihood  of  technical 
completion  for  commercial  R&D  projects  is  60%  [Bhat,  1991:262].  However,  since  the  DOE  is 
taking  a  long-term  view  with  its  applied  research  program,  it  is  appropriate  to  consider  the 
probability  that  a  technology  has  completed  its  development  as  a  function  of  time.  The  probability 
of  successful  development  is  then  the  probability  that  the  technology  in  question  has  been  released 
from  R&D  and  is  available  for  application.  This  “release  date”  is  defined  as  when  the  given 
technology  has  satisfied  all  of  its  specified  performance  criteria  and  is  considered  ready  for  use  in 
the  field. 
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Example  Technology  Selection  Decision 


Figure  1.1 


Basic  Methodology 


Th6se  utility  functions  relate  the  decision  maker's 
values  for  different  levels  of  P(dev)  &  P(use)  to 
the  specific  technologies  being  compared. 


Figure  1.2 
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Functional  Purpose 
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A  technology  will  therefore  be  considered  “successfully  developed”  after  this  release  date,  after 
the  point  when  the  technology  has  met  whatever  test  and  demonstration  standards  that  mark  the 
final  stage  of  R&D. 

1.2. 1.2  The  risk  of  successful  use,  P(use).  The  transfer  from  successful 
development  to  successful  implementation  is  a  step  whose  importance  is  often  underestimated. 
Even  if  a  technology  has  passed  all  of  its  developmental  test  and  evaluation  (DT&E) 
requirements,  there  is  still  no  guarantee  that  it  will  move  satisfactorily  to  the  field.  The  controlled 
environment  utilized  in  DT«&:E  rarely  represents  real-world  conditions.  Often  the  situations 
where  the  technology  is  put  to  use  are  different  from  those  anticipated  by  the  original  technology 
developers  [Leonard-Barton,  1987].  To  account  for  these  possibilities,  it  is  necessary  to  estimate 
the  likelihood  that  a  remediation  technology  is  successful  in  the  field,  given  that  it  was 
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successfully  developed  in  R&D.  This  “implementation  success”  depends  on  many  factors,  some 
site  dependent,  most  of  which  are  unknowable  at  the  present.  Nevertheless,  some  estimate  of  this 
implementation  success  is  arguably  the  most  important  criterion  in  judging  between  candidate 
technologies. 

1.2. 1.3  Independence  of  the  attributes.  By  defining  P(use)  as  the  probability  of 
successful  use  conditional  on  the  technology  being  first  “successfully  developed,”  we  can  consider 
the  two  attributes,  the  probabilities  of  development  and  implementation  success,  as  being 
independent.  P(use)  is  the  likelihood  that  the  technology  works  as  planned  once  it  has  completed 
R&D,  however  unlikely  it  is  that  that  technology  could  be  ready  for  use  within  the  given  time 
horizon.  By  accepting  the  assumption  that  the  test  and  demonstration  standards  which  a 
“successfully  developed”  technology  must  meet  remain  essentially  unchanged  through  its  multi¬ 
year  R&D,  we  can  say  that  its  P(use)  is  then  not  dependent  on  either  time  or  P(dev).  For  a  time 
horizon  farther  out  in  the  future,  one  would  expect  estimates  of  P(dev)  to  increase,  but  not 
P(use). 

1.2.2  Types  of  Technologies.  The  complete  remediation  process,  including  a  choice 
between  containment  and  removal- treatment-disposal  strategies,  is  shown  in  Figure  1.4.  This 
paradigm  of  landfill  remediation,  taken  fi-om  a  discussion  with  Dr  Jaffir  Mohuidden  in  August 
1995,  establishes  seven  possible  types  of  remediation  technologies  under  consideration  by  the 
DOE  Landfill  Stabilization  Focus  Area.  The  decision  maker  could  have  very  different  values  for 
the  same  levels  of  P(dev)  and  P(use)  between  two  types  of  technologies.  Potentially  the  decision 
maker  may  have  14  different  sets  of  preferences,  two  for  every  combination  of  technology  type. 
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1.2.2. 1  Characterization  and  Assessment.  These  technologies  identify,  quantify, 
and  locate  waste.  Characterization  of  the  site  is  necessary  to  determine  the  boundaries  of  the 
waste  and  identify  and  locate  specific  waste  forms  and  items,  such  as  drums,  large  metal  objects, 
and  voids.  Surveying  and  testing  results  in  a  site  contamination  map  which  is  used  to  plan  the  rest 
of  the  remediation  activities  [DOEa,  1995:xiv].  The  non-intrusive  characterization  of  waste 
containers  is  also  necessary  to  qualitatively  and  quantitatively  determine  the  waste  contents 
[DOEb,  1995:8]. 

1.2.2.2  Stabilization.  Stabilization  of  waste  in  situ  alters  the  waste’s  physical, 
chemical,  and  toxicological  properties  and  prevents  migration  of  waste  outside  the  site  boundaries 
[DOEa,  1995:xvi].  This  process  is  similar  to  containment  but  is  more  temporary  in  intention, 
delaying  the  need  for  treatment  and  allowing  for  different  remediation  alternatives  [Mohiudden, 
1995]. 


Remediation  Processes 

“technology  stream” 


passing  of  time 

Figure  1.4 
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1. 2.2.3  Removal.  Also  called  retrieval,  this  involves  excavation  of  waste  and 


contaminated  soil  for  ex  situ  treatment  and  disposal.  For  on-site  treatment,  treated 
decontaminated  soil  may  be  back-filled  instead  of  disposed  of  off-site.  Full-scale  removal 
typically  involves  conventional  drilling  and  excavation  equipment,  although  remotely  operated 
equipment  and  other  alternatives  are  being  developed  [DOEb,  1995;xiv-xv]. 

1. 2.2.4  Treatment.  There  are  both  ex  situ  and  in  situ  treatment  alternatives.  For 
ex  situ  technologies,  the  waste  debris  and  soil  is  packaged  and  transported  to  a  treatment  facility 
once  it  has  been  characterized  and  exhumed.  There  different  processes  will  immobilize,  detoxify, 
reduce  waste  volume,  and/or  stabilize  the  retrieved  waste  and  soil  for  disposal  on-site  or  at  an  off¬ 
site  storage  facility.  In  situ  treatment  methods  act  on  localized  and  isolated  waste  cells  to  treat  or 
stabilize  the  waste  in  place  [DOEa,  1995:xv;  DOEb,  1995:12-13]. 

1.2.2.5  Disposal.  After  treatment,  stabilized  final  waste  forms  that  are  not  back¬ 
filled  at  the  original  site  must  be  transported  to  another  site  for  disposal.  Disposal  technologies 
are  used  to  keep  the  treated  waste  isolated  from  the  environment  [DOEa,  1995:xv;  Mohuidden, 
1995]. 

1.2.2.6  Containment.  Containment  is  the  restriction  or  confinement  of  buried 
waste  to  a  limited  area  in  order  to  prevent  the  migration  or  leeching  of  waste  beyond  the 
confinement  boundaries.  Placement  of  various  barriers  around  the  waste  is  determined  by  site  and 
waste  characterization  data.  Containment  has  a  more  permanent  connotation  than  stabilization, 
although  long-term  containment  may  still  serve  as  an  interim  action  to  prevent  contaminant 
migration  pending  future  remediation  [DOEa,  1995 :xv;  Mohuidden,  1995]. 
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1. 2.2.7  Monitoring.  All  processes  use  monitoring  devices  to  assess  waste  being 
handled  and  watch  for  inadvertant  migration.  Long-term  monitoring  of  the  original  waste  site  and 
final  disposal  location  are  still  required  to  ensure  that  no  additional  remedial  action  is  needed  in 
the  future  [Mohuidden,  1995]. 

1.2.3  Utility  Curves  and  Preferences.  When  a  decision  maker’s  preferences  for  one 
attribute  are  examined,  for  example  P(dev),  a  utility  function  for  that  attribute  is  assessed.  A 
utility  (or  value)  function  for  a  decision  as  is  meant  here  is  a  scalar  index  of  a  decision  maker’s 
value  across  the  set  of  possible  alternatives.  The  utility  function  allows  the  comparison  of  various 
levels  of  the  attribute  with  respect  to  the  actual  value  that  the  decision  maker  places  on  those 
levels,  instead  of  the  levels  themselves.  In  this  case,  the  nominal  range  of  possible  alternatives 
goes  from  0%  to  100%.  A  utility  function  for  P(dev)  assigns  a  value  to  every  possible  estimate  of 
P(dev)  from  0%  to  100%  [Keeney,  1976:68].  This  value  is  the  “technical  risk  index”  score  for 
P(dev). 


Example  utility  curve 

assessed  for  attribute  P(use) 
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For  computational  reasons,  the  index  of  value  that  is  developed  from  the  utility  function  is 
normalized  between  0  and  1.  Of  all  the  possible  technologies,  one  with  a  P(dev)  of  0  will  be  least 
preferred  by  a  decision  maker.  That  least  preferred  alternative  is  assigned  a  “0"  utility.  A 
technology  with  a  P(dev)  of  1  will  be  most  preferred  and  so  has  a  utility  of  “1".  Once  these 
“anchors”  have  been  established,  the  remaining  values  then  must  be  assigned  for  the  other  possible 
estimates  for  P(dev)  between  0  and  1.  The  utility  function  must  increase  monotonically  (that  is, 
for  two  estimates  pi  and  p2  where  p2  >  pi,  the  value  of  p2,  U(p2),  must  always  be  greater  than 
or  equal  to  U(pl)).  An  example  of  a  utility  function  is  shown  in  Figure  1.5. 

1.2.4  Multi-Attribute  Utility  Theory.  Once  utility  functions  for  the  two  attributes  are 
assessed,  they  can  be  combined  to  form  an  overall  value  score  for  that  specific  alternative.  If  the 
utilities  have  been  assessed  and  combined  correctly,  the  ordering  of  a  set  of  alternatives  based  on 
their  utilities  will  be  the  same  as  the  order  the  decision  maker  actually  prefers  (so  long  as  the 
decision  maker  remains  consistent  in  his  preferences). 

The  heart  of  the  matter  is  then  to  combine  the  two  single-attribute  utility  functions  into 
one  final  value  score  or  index  allows  us  to  see  exactly  the  sorts  of  trade-offs  the  decision  maker  is 
willing  to  make  to  get  the  best  alternative  available.  This  establishes  what  combinations  of  the 
two  attributes  to  which  the  decision  maker  is  indifferent.  While  more  complicated  functional 
forms  exist,  two  simple  functions,  the  additive  and  multilinear  utility  functions,  sufficiently 
represent  decision  maker  preferences  in  most  cases  [Stewart,  1995:257].  Both  will  be  used  in  this 
work. 
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1.2.4. 1  Additive  utility  function.  The  additive  utility  function  simply  adds  the  two 
single  attribute  utility  functions  after  weighting  them  by  appropriate  weights.  For  our  two 
attributes  P(dev)  and  P(use)  (abbreviating  them  as  &  P„  for  brevity),  the  combined  additive 
utility  function  is: 


where  Ud(P^)  is  the  single  attribute  utility  function  for  P(dev)  and  Uu(PJ  is  the  utility  function  for 
P(use).  The  two  weights,  and  k„,  must  sum  to  one  for  the  final  combined  utility  to  remain 
between  0  and  1.  So  k^  =  l  -  [Clemen, 199 1:480-4].  The  values  of  and  indicate  the 
relative  importance  or  weight  the  decision  maker  places  on  the  two  attributes  P(dev)  and  P(use). 
For  example,  if  k^  <  P(use)  is  considered  more  important.  To  use  this  multi-attribute  utility 
function,  only  a  single  constant,  must  be  assessed  in  addition  to  the  original  utility  functions  for 
P(dev)  and  P(use). 

This  type  of  multi-attribute  utility  function  implies  that  P(dev)  is  utility  independent  of 
P(use)  (and  vice  versa).  That  is,  conditional  preferences  for  estimates  of  P(dev)  do  not  depend  on 
the  particular  level  of  P(use).  Put  another  way,  the  decision  maker’s  preference  between  the 
alternative  technologies  A  and  B,  A  having  (P(dev),  P(use))  estimates  of  (pi,  pO)  and  B  having 
(p2,  pO),  only  depends  on  pi  and  p2  and  does  not  change  for  any  level  of  pO.  This  utility 
independence  is  the  characteristic  which  allows  us  to  meaningfully  use  single-attribute  utility 
functions  in  the  first  place  [Keeney, 1976:226]. 
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The  additive  utility  function  also  implies  that  P(dev)  and  P(use)  are  additive  independent 
[Keeney,  1976:229-32].  Consider  the  following  example,  to  illustrate  the  concept  for  our 
decision  problem.  Suppose  we  have  four  possible  candidate  technologies:  A  with  (pi,  p3),  B 
with  (p2,  p3),  C  with  (pi,  p4),  and  D  with  (p2,  p4),  where  pi  <  p2  and  p3  <  p4.  When 
comparing  technologies  A  and  B,  our  decision  maker  would  clearly  prefer  B  because  pi  <  p2. 
Likewise  he  or  she  would  prefer  D  over  C.  Additive  independence  means  our  decision  maker 
would  prefer  B  over  A  the  same  amount  as  D  is  preferred  over  C,  measured  in  utility  —  that  is, 
U(p2,  p3)  -  U(pl,  p3)  =  U(p2,  p4)  -  (pi,  p4). 

Put  another  way,  additive  independence  allows  for  no  interaction  between  the  decision 
maker’s  preferences  for  various  levels  of  P(dev)  and  P(use).  For  example,  under  additive 
independence  a  technology  with  a  P(dev)  of  .9  will  always  be  preferred  to  one  with  a  P(dev)  of 
0.8  in  the  same  way  whether  both  technologies  have  a  P(use)  of  0. 1  or  0.95.  While  this  condition 
is  not  trae  for  most  decision  makers,  often  an  additive  utility  function  does  well  enough  in 
modeling  their  preferences  to  justify  using  it  [Clemen, 1991:483;  Stewart,  1995]. 

1.2.4.2  Multilinear  utility  function.  The  multilinear  utility  function  is  a 
generalization  of  the  additive  function,  where  the  requirement  for  additive  utility  independence  is 
relaxed  and  interactions  between  the  attributes  is  allowed.  Mutual  utility  independence  is  still 
required,  however.  The  form  of  the  function  is: 

U(P„  PJ  -  )  (2) 
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using  the  same  notation  as  with  equation  1.  Note  that  the  only  differences  are  that  l)k^*  I- 
and  2)  an  interaction  term  has  been  added  [Keeney, 1976:233].  The  impact  of  the  interaction  term 
is  set  by  the  two  constants,  and  k^.  The  greater  the  magnitude  of  the  (1  -  k^  term,  the 
greater  the  effect  of  the  interaction.  If  the  (1  -  k^  term  is  zero,  the  multilinear  function 
defaults  to  the  additive  utility  function  already  discussed  [Clemen,1991:481]. 

When  assessing  a  multilinear  utility  function,  only  one  more  step  is  needed  beyond  the 
assessment  of  an  additive  function.  An  additional  trade-off  between  two  alternative  technologies 
is  made,  where  one  alternative  is  set  with  a  given  (P(dev),  P(use))  estimate  while  the  other’s 
attributes  are  allowed  to  vary.  The  decision  maker  is  asked  to  set  the  second  technology’s  P(dev) 
and  P(use)  at  a  different  pair  of  levels  where  he  or  she  would  be  indifferent  between  the  two 
candidate  technologies.  The  answer  to  this  question  provides  the  necessary  additional  information 
to  establish  the  multilinear  utility  function. 

1. 2.4.2. 1  Substitutes  and  Complements.  The  interpretation  of  this 
additional  term  (1  -  is  not  as  straightforward  as  that  of  the  two  weights  ^^and  k„.  If  this 
term  is  positive,  high  values  of  P(dev)  and  P(use)  raise  the  overall  utility  higher  than  the  two 
additive  terms  alone.  Then  the  attributes  are  called  compliments  of  each  other.  A  low  estimate  of 
one  will  severely  reduce  the  combined  overall  utility.  One  low  estimate  is  almost  as  bad  as  having 
two,  and  a  high  utility  is  only  found  by  high  estimates  of  both.  If  (1  -  k^-  kj  was  negative,  the 
two  attributes  are  substitutes  for  one  another.  Here  high  values  of  both  result  in  a  large  product 
term  that  is  subtracted  from  the  additive  total.  In  a  sense,  preferred  values  of  P(dev)  and  P(use) 
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work  against  each  other.  But  when  one  is  high  and  the  other  low,  the  subtracted  term  is  not  very 
large.  This  means  the  overall  utility  can  still  be  reasonably  high  despite  one  attribute  being  low. 


Example  Indifference  Curves 

C’s  Surface  (contours),  Estimate  4 


Figure  1.6 


1.2.5  Indifference  Curves.  Once  the  multi-attribute  utility  function  has  been  assessed,  the 
mathematical  model  of  the  decision  maker’s  preferences  can  now  be  expressed.  One  tool  for 
looking  at  these  preferences  are  indifference  curves.  These  curves  are  plotted  on  a  graph  with 
P(dev)  on  one  axis  and  P(use)  on  the  other,  and  show  the  sets  of  technologies  whose  total 
combined  utility  is  the  same.  The  decision  maker  wiU  be  indifferent  between  any  technologies 
whose  P(dev)  and  P(use)  attributes  place  them  on  the  same  line  on  the  basis  of  the  two  technical 
risk  factors.  See  Figure  1.6  for  an  example  where  different  possible  alternatives  (T1  through  TIO) 
are  graphed  along  with  the  indifference  curves.  Each  of  the  lines,  going  from  lower  left  to  the 
upper  right,  represents  utilities  of  0.1, 0.2, 0.3,  and  so  on.  Graphing  indifference  curves  provides 
insight  into  the  decision  maker’s  trade-offs  between  the  two  attributes  [Clemen, 199 1:442]. 
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1.2.6  Utility  Surfaces.  Plotting  the  combined  utility  as  a  three-dimensional  surface  above 
the  plane  provided  by  the  two  attributes  as  axes  can  provide  another  visualization  of  the  decision 
maker’s  preferences.  These  surfaces  are  closely  related  to  the  indifference  curves  introduced 
above  —  the  indifference  curves  are  merely  the  contours  of  the  utility  surface.  These  three- 
dimensional  graphs  are  the  most  clear  representation  of  complex  preference  relationships  between 
two  attributes.  See  Figure  1.7  for  an  example.  Each  color  represents  a  band  of  utilities,  climbing 
from  0  to  0.2  (blue),  0.2  to  0.4  (green),  and  so  on. 


Example  Utility  Surface 


Figure  1.7 


1.2.7  Assessment  Questions.  The  first  issue  that  was  explored  in  the  interviews  with  the 
three  “decision  makers”  was  the  degree  of  additive  and  utility  independence  between  P(dev)  and 
P(use)  for  that  person  (this  was  done  using  an  adaptation  of  a  dialog  in  Keeney  and  Raiffa 
[1976:299-300]).  As  already  mentioned,  there  is  some  tolerance  for  not  strictly  adhering  to  the 
requirements  of  additive  and  utility  independence. 
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The  interviewees  were  then  asked  questions  first  to  establish  their  values  across  the 
possible  ranges  of  P(dev)  and  P(use)  and  then  to  set  the  weights  for  their  combined  utility 
function.  The  interviews  took  from  45  minutes  to  two  and  a  half  hours  to  conduct,  depending  on 
the  complexity  of  the  interviewee’s  preferences.  The  dialog  between  interviewer 
andintervieweewas  a  learning  process  for  both  participants  and  proceeded  quickly  once  both  were 
comfortable  with  the  assessment  process  and  the  interviewee’s  values. 


Utility  Function  Assessment  Template 


remediation  process: 
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Figure  1.8 


Assessing  the  single  utility  functions  was  done  graphically,  using  the  template  shown  in 
Figure  1.8.  The  technology  type  and  attribute  that  was  being  assessed  were  first  recorded  (for 
example,  “treatment  technologies”  and  “P(dev)”).  Often  utility  functions  were  the  same  for 
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several  technology  types,  making  the  assessment  process  simpler.  The  interviewer  and 
interviewee  then  agreed  that  the  highest  value  (utility  =  1)  should  be  given  to  technologies  which 
were  certain  to  be  available  or  be  successfully  used  (P(success)  =  100%).  The  interviewee  was 
asked  how  much  he  or  she  would  be  willing  to  pay  to  buy  such  a  technology,  to  reinforce  a  value- 
focused  perspective.  A  lower  limit  of  value  would  then  be  set  for  technologies  that  had  no  chance 
of  successful  development  or  use  (P(success)  =  0). 

Once  the  endpoints  of  the  value  function  were  set,  the  interviewee  was  asked  how  much 
he  would  pay  for  a  technology  with  only  a  50-50  chance  of  being  successfully  developed  or  used. 
This  corresponded  to  the  utility  of  P(success)  =  50%.  Similar  questions  were  asked  to  find  other 
points  on  the  utility  function  until  the  interviewee  was  satisfied  with  the  shape.  Once  the 
interviewee  had  become  familiar  with  this  process,  he  could  draw  his  value  curves  directly  without 
much  prompting. 

After  utility  functions  were  assessed  for  aU  combinations  of  technology  type  and  attribute, 
the  interviewee  was  asked  questions  to  determine  the  constants  of  his  multi-attribute  utility 
function.  A  computer  software  program.  Logical  Decisions®,  was  used  to  frame  the  necessary 
questions.  For  those  interviewees  whose  combined  utility  function  was  additive,  a  simple  pair  of 
bars  were  used  to  establish  relative  weights  between  P(dev)  and  P(use).  The  decision  maker 
could  change  the  lengths  of  these  bars  until  he  felt  they  reflected  the  relative  importance  of  the 
two  attributes.  For  those  with  multilinear  combined  utility  functions,  one  more  question  was 
asked  to  establish  a  trade-off  as  described  in  section  1. 2.4.2.  Logical  Decisions®  calculated  the 
values  of  the  constants  from  the  answers  to  these  questions. 
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The  Logical  Decisions®  software  was  also  used  after  the  interviews  were  finished,  to 
reproduce  the  hand-drawn  single-attribute  utility  curves  of  the  interviewees  and  determine  the 
equations  of  the  curves.  These  equations  were  then  used  in  a  spreadsheet  to  produce  the  graphs 
that  follow.  The  final  indifference  curves  and  utility  surface  graphs  in  section  5.0  were  produced 
using  Matlab®. 
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11.  Preferences  of  the  Example  Decision  Makers 


2.1  The  “Decision  Makers” 

The  methods  described  in  the  previous  section  can  model  the  preferences  of  any  waste 
remediation  program  manager  to  support  technology  investment  choices.  To  illustrate  their  use, 
three  experienced  people  working  with  the  Landfill  Stabilization  Focus  Area  Field  Office  at  the 
Savannah  River  site  were  interviewed  and  their  utilities  assessed.  While  these  interviewees  were 
not  actual  DOE  managers,  they  each  had  experience  working  with  these  issues  and  supporting  the 
Department  of  Energy.  The  three  individuals  each  had  a  very  different  focus  on  what  they 
considered  most  relevant  for  technology  decisions.  Utility  functions  for  all  the  types  of 
remediation  technology  were  assessed  and  are  graphed  below. 

Since  the  identities  of  the  interviewees  are  irrelevant  for  this  report,  they  are  labeled  “A,” 
“B,”and“C.” 

The  time  horizon  used  to  frame  the  technology  selection  preferences  for  P(dev)  was  seven 
years  from  now  (Dec  1995),  as  selected  by  Dr  Paul  Krumrine.  The  utility  functions  assessed  for 
P(dev)  are  only  valid  for  this  time  frame.  Additional  assessment  would  be  required  to  examine 
different  horizons  for  technology  investment  preferences. 

2.2  A’ s  Preferences 

Interviewee  A  had  a  very  simple  perspective  of  a  technology’s  value  to  the  Department  of 
Energy.  He  believed  that  no  technology  was  worth  anything  to  DOE  unless  it  had  a  predicted 
75%  chance  of  being  ready  within  seven  years  or  an  predicted  75%  chance  of  successful 
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A  s  values  for  P(dev) 

all  technologies 


Figure  2. 1 


A's  combined  preferences 

for  all  technologies 


Utility 


A's  values  for  P(use) 

all  technologies 


A's  combined  preferences 

for  all  technologies  --  another  view 


implementation  in  the  field.  The  pressure  from  complying  with  current  remediation  agreements 
and  budget  constraints  would  leave  DOE  with  no  other  options  but  the  more  certain  technologies. 

The  following  graphs,  Figures  2.1  and  2.2,  show  his  two  utility  functions  for  all 
technology  types.  His  preferences  for  P(dev)  and  P(use)  place  no  value  on  technologies  with  less 
than  an  estimated  75%  chance  of  being  available  or  working  as  expected  (the  slope  of  the 
increase  in  value  at  75%  in  both  graphs  is  an  artifact  of  the  graphing  software  —  the  actual  utility 
function  has  a  discontinuity  at  75%,  where  the  utility  goes  from  0  to  0.5).  Above  his  75% 
threshold,  however,  he  felt  that  his  values  scaled  linearly  with  increasing  likehhood  of  success. 

A  had  an  additive  multi-attribute  utility  function  and  placed  more  value  on  P(use)  than  on 
P(dev).  His  utility  surface  is  graphed  in  figures  2.3  and  2.4.  This  “floor”  of  zero  value  for  any 
technology  that  has  an  estimate  beneath  his  0.75  threshold  shows  how  he  would  exclude  anything 
but  the  minority  of  proven  and  nearly  developed  remediation  technologies  from  consideration. 

2.3  B’ s  Preferences 

B  had  recent  experience  dealing  with  actual  waste  site  management  and  so  had  more  of  a 
landfill  owner’s  perspective.  He  proved  to  have  the  most  complex  set  of  preferences,  with  a 
different  utility  function  for  almost  every  technology  type.  P(dev)  and  P(use)  were  not  additively 
independent  for  B,  and  so  a  multilinear  combined  utility  function  was  used. 

B  did  not  agree  with  the  remediation  process  paradigm  shown  in  Figure  1.4.  He  believed 
that  the  monitoring  process  could  actually  be  one  of  the  first  stages,  since  other  processes  would 
only  be  needed  at  a  landfill  if  waste  was  detected  migrating  out  of  it.  This  disagreement  did  not 
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invalidate  either  the  approach  to  technical  risk  or  the  method  of  comparing  technologies,  since 
technologies  should  only  be  compared  against  others  from  the  same  process. 

He  also  disagreed  with  the  interviewer’s  assumption  that  a  technology  with  zero  chance  of 
successful  implementation  would  cost  nothing  (i.e.  0  utility  =  0  cost),  and  correctly  pointed  out 
that  failure  in  the  field  for  critical  processes  could  have  negative  value  to  the  site  owners.  In  a  few 
cases  he  set  the  lower  limit  of  his  values  for  P(use)  at  a  negative  dollar  value,  with  respect  to  the 
amount  he  would  be  willing  to  spend  to  buy  a  technology  certain  to  succeed.  This  did  not  present 
any  difficulty.  These  ranges  of  value  were  normalized  to  a  0  to  1  interval  afterward  to  relate  his 
values  to  a  standard  (0,1)  utility,  where  his  lowest  negative  dollar  cost  (for  P(use)  =  0)  was 
assigned  zero  utility  and  the  rest  were  scaled  up  from  there.  It  is  the  shape  of  his  value  curve  that 
is  important,  and  was  preserved. 

B’s  single  attribute  utility  functions  are  graphed  below,  in  Figures  2.5  to  2.13.  He 
grouped  characterization,  containment,  stabilization,  and  removal  technologies  into  one  set  with 
the  same  value  function  for  P(dev).  He  also  considered  treatment  and  disposal  technologies  as 
having  the  same  value  function  for  both  P(dev)  and  P(use).  All  other  technologies  were  regarded 
individually  by  B. 

B’s  utility  functions  are  generally  in  the  shape  of  S-curves,  with  some  lower  threshold  of 
estimated  success  required  before  any  significant  value  was  placed  on  a  technology  climbing  up  to 
some  upper  threshold  beyond  which  he  is  relatively  indifferent.  Unlike  A,  however,  B  believed 
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utility  utility  utility 


B's  values  for  P(dev) 

for  characterization,  containment,  stabilization,  & 


B's  values  for  P(clev) 

treatment  &  disposal  technologies 


B's  values  for  P(dev) 

for  monitoring  technologies 


B's  values  for  P(use) 

for  characterization  technologies 


B's  values  for  P(use) 

for  stabilization  technologies 


Figure  2.9 


B's  values  for  P(use) 

for  containment  technologies 
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B's  values  for  P(use)  B's  values  for  P(use) 

for  removal  technologies  treatment  &  disposal  technologies 


B's  values  for  P(use) 


for  monitoring  technologies 


there  was  some  value  to  working  on  technologies  with  little  chance  of  successful  development, 
depending  on  the  type  of  technology. 

B  was  more  tolerant  of  lower  likelihoods  of  successful  development  with  treatment, 
disposal,  and  monitoring  technologies.  This  is  due  to  his  beliefs  that  adequate  methods  for  these 
processes  already  exist  and,  in  the  cases  of  treatment  and  disposal,  that  the  processes  themselves 
were  not  that  crucial.  For  monitoring  technologies,  his  utility  function  is  simply  linear  (see  Figure 
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2.7).  Otherwise  he  placed  nearly  equal  value  on  technologies  with  a  P(dev)  of  between  75%  to 
100%,  with  a  steep  drop-off  in  value  below  that  range  (see  Figure  2.5). 

B’s  detailed  preferences  for  P(use)  are  very  interesting.  He  believed  monitoring 
technologies  to  be  the  most  important  with  respect  to  successful  use,  since  any  failures  in  the  field 
could  lead  to  neglecting  waste  migration  and  more  difficult  (and  costly)  remediation  needs. 
Almost  as  important  are  successful  removal  technologies,  since  that  is  the  most  crucial  step  in  the 
removal-treatment-disposal  strategy  for  the  owner  of  a  waste  site  (since  after  removal  the  waste  is 
out  of  the  landfill  and  poses  a  much  lower  threat  of  contamination  at  the  site).  For  both  these 
types  of  technologies,  B’s  utility  curves  give  high  value  only  to  relatively  high  estimates  of  P(use) 
(see  Figures  2.13  and  2.11,  respectively).  Stabilization  and  containment  technologies  had  value 
functions  more  tolerant  of  lower  likelihoods  of  success,  since  even  partial  successes  would  slow 
the  leakage  of  waste  from  the  site.  B  also  felt  that  successful  detailed  characterization  of  the 
landfiH’s  waste  streams  was  not  as  significant,  since  successful  removal  and  containment 
approaches  did  not  necessarily  depend  on  it. 

B’s  utility  function  for  treatment  and  disposal  technologies  has  the  most  unusual  shape  — 
unlike  the  rest,  it  is  just  a  simple  concave  exponential  utility  curve  (see  Figure  2.12).  This  fits  his 
view  that  treatment  and  disposal  are  the  least  important  remediation  processes.  Less  than 
complete  success  here  means  little  to  the  landfill  manager,  since  the  waste  is  no  longer  his 
responsibility. 

B’s  multiattribute  utility  surfaces  are  all  multilinear  in  form.  He  placed  far  more  value  on 
P(use)  than  on  P(dev),  consistent  with  his  previous  experience.  His  additional  trade-off 
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utility  utility 


B's  combined  preferences 

for  characterization  technologies 


Figure  2.14 


B's  combined  preferences 

for  Stabilization  technologies 


Figure  2.15 


Figure  2.16 


B's  combined  preferences 

for  removal  technologies 


Figure  2.17 
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B's  combined  preferences  B’s  combined  preferences 

for  treatment  &  disposal  technologies  for  monitoring  technologies 


Figure  2.18  Figure  2.19 


established  the  relationship  between  the  single-attribute  utility  functions  and  the  constants  kj  and 
k„.  His  various  utility  surfaces  follow  in  Figures  2.14  through  2.19. 

The  small  orange  areas  (total  utility  >  0.8)  show  B’s  opinions  about  the  importance  of 
monitoring  and  removal  technologies  (Figures  2.19  and  2.17,  respectively).  Only  technologies 
with  high  P(dev)  and  P(use)  estimates  have  high  value  to  him  in  these  crucial  processes. 
Likewise,  the  gradual  slope  and  broad  orange  area  of  B’s  surface  for  treatment  and  disposal 
technologies  show  how  relatively  indifferent  he  is  to  these  processes  (see  Figure  2. 1 8).  One 
interesting  thing  to  notice  is  how  four  out  of  B’s  surfaces  (fig.  2.14-2.17)  resemble  A’s 
discontinuous  ones  to  some  degree,  despite  having  drastically  different  single-attribute  utility 
curves. 
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2.4  C  s  Preferences 

Unlike  A’s  programmatic  perspective  or  B’s  site  management  one,  C  had  more  of  a 
technology  developer’s  viewpoint.  His  experience  in  R&D  for  waste  remediation  was  the  source 
of  his  preferences.  Figures  2.20  to  2.22  show  his  single  attribute  utility  functions. 

C  has  a  vaguely  S-curve  utility  function  for  P(dev),  as  is  shown  in  Figure  2.20.  He  also 
disagreed  with  the  interviewer’s  assumption  that  a  decision  maker  would  want  to  spend  zero 


C's  values  for  P(dev) 

for  all  technologies 


C's  values  for  Plusel 

characterization,  stabilization,  containment. 


Figure  2.21 


C's  values  for  P(use) 

removal,  treatment,  &  disposal  tech. 


Figure  2.22 


2-10 


funds  on  technologies  with  no  chance  of  being  developed  within  the  time  horizon  —  he  preferred 
to  spend  at  least  5%  of  what  he  would  pay  for  technologies  certain  to  be  ready  on  development 
efforts  that  would  only  pay  off  in  the  long  term.  His  value  curves  were  rescaled  to  fit  a  (0,1) 
standard  utility  range,  just  as  B’s  were.  Like  B,  C  felt  some  remediation  processes  required 
different  utility  curves  for  P(use).  He  thought  removal,  treatment,  and  disposal  technologies  were 
too  important  to  give  much  value  to  alternatives  with  poor  estimated  field  performance  (see 
Figure  2.22).  He  was  more  tolerant  of  less  robust  alternatives  in  the  other  remediation  processes 
(see  Figure  2.21). 


C's  combined  preferences 

for  characterization,  stabilization,  containment,  &  monitoring  tech. 


C's  combined  preferences 


rennoval,  treatment,  &  disposal  tech. 


Figure  2.23 


Figure  2.24 


C,  like  B  and  A,  thought  P(use)  was  more  important  than  P(dev).  The  differences 
between  the  curves  in  Figures  2.21  and  2.22  have  a  impact  on  the  final  utility  surfaces  for  the  two 
groups  of  technology  types.  C’s  relative  intolerance  for  failure  in  the  field  for  removal,  treatment, 
and  disposal  technologies  is  evident  in  the  greater  slope  of  the  surface  in  Figure  2.24.  There  is  a 
smaller  area  of  utility  >  .8  than  in  his  other  surface  for  characterization,  stabilization,  containment, 
and  monitoring  technologies  (see  Figure  2.23). 
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In  general,  C’s  surfaces  show  higher  value  for  candidate  technologies  with  high  P(use)  and 


low  P(dev)  estimates  than  the  others.  This  reflects  his  views  about  funding  technologies  with 
higher  development  risk. 
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III.  Candidate  Treatment  Technologies 


3.1  Example  Technology  Selection  Problem 

It  was  decided  to  exercise  the  models  of  A,  B,  and  C’s  preferences  with  an  example 
decision  problem  of  the  sort  faced  by  DOE  technology  managers.  A  recent  review  of  a  group  of 
treatment  technologies  being  developed  through  the  DOE  had  just  been  conducted.  Dr  Paul 
Krumrine  was  able  to  ask  a  few  attendees  of  this  review  to  estimate  the  two  technical  risk  factors 
for  ten  of  these  technologies.  The  definitions  used  in  this  report  for  probabilities  of  successful 
development  and  successful  field  use  were  explained  and  a  time  horizon  of  seven  years  was  set. 
Dr  Krumrine  faxed  these  estimates  to  AFIT/ENS  in  December  1995. 

By  using  the  utihty  functions  assessed  for  A,  B,  and  C  on  treatment  technologies,  a 
ranking  of  the  ten  technologies  based  on  the  “decision  makers’”  preferences  for  these  technical 
risk  factors  can  be  generated.  Analysis  of  the  results  will  help  evaluate  the  usefulness  of  this 
approach. 

3.2  Technical  Risk  Estimates 

The  estimates  of  P(dev)  and  P(use),  designated  1  through  4,  were  made  by  four  experts 
equipped  with  our  definitions  of  P(dev)  and  P(use)  for  ten  different  treatment  technologies 
currently  under  development  by  the  Department  of  Energy.  Hereafter  we  will  refer  to  expert  1  as 
providing  estimate  1,  and  so  on.  The  candidate  technologies  are  listed  in  Table  3.1  with  the 
corresponding  P(dev)  and  P(use)  estimates.  The  table  should  be  read,  for  Joule  Heated  Melters 
as  an  example,  as  saying  expert  1  assessed  the  probability  of  this  technology  being  ready  within 


3-1 


seven  years  as  100%  and  the  probability  of  successful  field  use  as  75%.  The  codes  are  supplied 
for  interpreting  the  graphs  that  follow. 


P(dev)  &  P(use)  Estimates  for  10  Treatment  Technologies 


P(dev)  estimated  for  availability  by  7  years  from  now 


estimate  1 

estimate  2 

estimate  3 

estimate  4 

code 

P(dev 

P(use) 

P(dev 

P(use) 

P(dev 

P(use) 

P(dev 

P(use) 

) 

) 

) 

) 

Joule  Heated  Mclters 

T1 

100 

75 

95 

95 

80 

75 

100 

100 

DC  Graphite  Plasma  Ai*c 

T2 

75 

90 

90 

60 

100 

AC  Arc  Melters 

T3 

1^^ 

50 

50 

35 

50 

■^1 

80 

In  Situ  Vitrification 

T4 

75 

25 

70 

60 

70 

60 

100 

100 

Russian  Hi-Frequency 

T5 

75 

25 

20 

30 

10 

10 

80 

100 

Induction 

Russian  Hybrid  Plasma 

T6 

50 

25 

20 

50 

15 

15 

30 

50 

Plasma  Hearth  Process 

T7 

25 

25 

50 

40 

40 

30 

20 

50 

Plasma  Centrifical  Furnace 

T8 

25 

15 

50 

40 

25 

20 

20 

20 

Vortec  Combustion 

T9 

10 

10 

70 

20 

10 

10 

20 

20 

Microwave  Melters 

TIO 

10 

10 

20 

10 

10 

10 

20 

20 

Table  3.1 


The  four  following  graphs,  Figures  3.1  to  3.4,  depict  each  set  of  estimates  in  a  different 
form.  These  plots  will  be  superimposed  on  the  utility  surfaces  of  A,  B,  and  C  corresponding  to 
treatment  technologies  in  section  4.0  to  graphically  illustrate  how  these  utilities  generate  rankings 
of  the  candidates.  Please  refer  back  to  Figures  3.1  through  3.4  to  help  interpret  the  surface  plots. 
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Note  that  several  technologies  have  the  same  estimated  P(dev)  and  P(use)  and  so  should  have  the 
same  resulting  combined  utilities. 


Technology  Estimate  2 

Mlimilid  a  P(ui4,  iiyyn 


Technology  Estimate  1 

estimated  P(dev)  &  P(use),  at  7  yrs 


Figure  3. 1 


Figure  3.2 
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Technology  Estimate  3 

estimated  P(dev)  &  P(use),  at  7  yrs 


0  0.2  0.4  0.6  0.8 

P(dev) 


Figure  3.4  Figure  3.3 
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IV.  Results  of  Sample  Decision  Problem 
4.1  Summary  of  Resulting  Utilities 

Table  4.1  lists  the  resulting  utilities  of  the  ten  treatment  technologies  for  the  12 
combinations  of  “decision  maker”  and  technical  risk  estimate. 


Combined  Utilities 

for  each  combination  of  preference  and  expert  estimate  of  risk  parameters 


estimate: 

A’s  preferences 

B’s  preferences 

C’s  preferences  || 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 

Joule  Heated 
Melters 

.616 

.9 

.523 

1.00 

.944 

.989 

.931 

1.00 

.55 

.945 

.501 

1.00 

DC  Graphite 
Plasma  Arc 

.5 

.8 

0 

.907 

.927 

.974 

.745 

.988 

.484 

.866 

.125 

.951 

AC  Arc  Melters 

0 

0 

0 

0 

.780 

.874 

.684 

.898 

.082 

.259 

.067 

.441 

In  Situ  Vitrification 

0 

.877 

0 

1.00 

.656 

.986 

.868 

1.00 

.185 

.934 

.241 

1.00 

Russian  Hi- 
Frequency 
Induction 

0 

0 

0 

.907 

.656 

.482 

.198 

.988 

.185 

.014 

.186 

.951 

Russian  Hybrid 
Plasma 

0 

0 

0 

0 

.600 

.635 

.289 

.694 

.066 

.029 

.006 

.041 

Plasma  Hearth 
Process 

0 

0 

0 

0 

.469 

.720 

.602 

.635 

.019 

.069 

.043 

.029 

Plasma  Centrifical 
Furnace 

0 

0 

0 

0 

.353 

.720 

.414 

.377 

.019 

.069 

.019 

.013 

Vortec  Combustion 

0 

0 

0 

0 

.198 

.601 

.198 

.377 

.003 

.166 

.003 

.013 

Microwave  Melters 

0 

0 

0 

0 

.198 

.245 

.198 

.377 

.003 

.013 

.003 

.013 

Table  4.1 
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The  very  different  utility  siufaces  for  A,  B,  and  C  shown  in  section  III  suggested  that  the 
combined  utilities  for  treatment  technologies  would  vary  widely  between  the  “decision  makers.” 
The  results  in  Table  4.1  show  this  impression  was  correct.  While  those  technologies  with 
generally  high  estimates  for  both  P(dev)  and  P(use)  all  tend  to  have  utilities  of  0.9  or  more,  the 
others’  utilities  differ  considerably.  C’s  utilities  are  all  much  lower  than  B’s  in  these  ranges  of 
(P(dev),  P(use))  pairs  for  the  same  estimate,  sometimes  by  more  than  an  order  of  magnitude.  If 
one  excluded  all  technologies  whose  utilities  were  less  than  0.2,  the  set  of  these  eliminated 
technologies  would  look  very  similar  for  both  A  and  C.  This  suggests  more  parallels  between  A 
and  C’s  values  than  their  utility  surfaces  suggest 

Table  4.2  shows  the  resulting  rank  orderings  of  the  different  technologies  organized  by 
"decision  maker,"  while  Table  4.3  is  organized  by  expert  estimate.  The  in  the  columns  of  A’s 
results  reflect  the  zero  utilities  generated  by  his  utility  function.  Since  there  were  many 
technologies  whose  attributes  were  estimated  to  be  the  same,  there  are  ties  inside  several  sets  of 
rankings. 

These  rankings  are  surprising  in  their  similarities,  despite  the  differences  in  utilities  from 
Table  4.1.  Reviewing  first  rankings  in  the  same  estimate  set,  the  rank  ordering  of  technologies  is 
consistently  the  same  for  the  different  “decision  makers.”  The  rankings  are  identical  for  aU  three 
“decision  makers”  in  both  estimate  3  and  estimate  4,  and  only  differ  by  a  few  interchanged 
positions  in  estimates  1  and  2.  This  suggests  that  the  three  “decision  makers”  were  actually  very 
consistent  with  each  other’s  values,  despite  their  different  orientations  (A  -  programmatic,  B  - 
waste  site  ownership,  &  C  -  technology  development). 
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Rankings  By  ‘‘Decision  Maker” 


for  each  combination  of  preference  and  expert  estimate  of  risk  parameters 


estimate: 

A’s  preferences 

B’s  preferences 

C’s  preferences 

1 

2 

3 

4 

1 

2 

3 

4 

1 

2 

3 

4 
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Tl 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

DC  Graphite  Plasma  Arc 

T2 

2 

3 

- 

3 

2 

3 

3 

3 

2 

3 

3 

3 

AC  Arc  Metiers 

T3 

■ 

B 

B 

B 

3 

B 

B 
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5 

B 

B 

5 

til  Situ  Vitrification 

T4 
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2 

B 

1 

4 

2 

2 

1 

3 

1 

Russian  Hi-Frequency 
Induction 

T5 

1 

1 

■ 

3 

1 

9 

8 

3 

3 

9 

8 

3 

Russian  Hybrid  Plasma 

T6 

B 

B 

-- 

B 

6 

7 

7 

6 

6 

8 

7 

6 

Plasma  Hearth  Process 

T7 

B 

B 

B 

B 

7 

5 

5 

7 

7 

6 

5 

7 

Plasma  Centrifical  Furnace 

T8 

B 

B 

B 

B 

8 

5 

6 

8 

7 

6 

6 

8 

Vortec  Combustion 

T9 

B 

B 

B 

B 

9 

8 

8 

8 

9 

5 

8 

8 

Microwave  Metiers 

TIO 

-- 

-- 

--  1 

9 

10 

8 

8 

9 

10 

8 

8 

Table  4.2 


In  review,  the  rankings  by  preference  set  in  Table  4.2  show  that  the  top  four  technologies 
remained  consistently  Tl,  T2,  T3,  and  T4.  Only  once  is  T3  pushed  back  to  #5  (C,  estimate  1) 
because  for  that  estimate  and  preference  set  T5  is  valued  more.  The  bottom  technologies  also 
stay  mostly  the  same  between  B  and  C  across  the  different  estimates. 

The  only  technology  which  changes  place  often  is  the  Russian  Hi-Frequency  Induction 
method,  T5.  In  some  cases,  T5  is  one  of  the  least  preferred  technologies,  while  in  others  it  is  in 
the  top  three  or  four.  Its  rank  stays  mostly  consistent  between  “decision  makers”  inside  one 
estimate,  which  suggests  the  different  rankings  has  to  do  with  very  different  risk  estimates  by  the 
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four  experts  (a  glance  at  Table  3.1  supports  this).  The  ranking  of  T5  will  be  discussed  in  a  later 
section. 


Rankings  By  Expert 


for  each  combination  of  expert  estimate  and  “decision  maker”  preference 


expert  1 

expert  2 

expert  3 

expert  4  || 
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■ 
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T6 

B 

6 

6 

7 

8 
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T8 

1 
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1 

■ 
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■ 
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Vortec  Combustion 

T9 
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9 

9 

B 

8 

5 

B 

8 

8 

B 

8 

8 

Microwave  Melters 

TIO 

9 

9 

□ 

10 

10 

□ 

8 

8 

□ 

8 

8 

Table  4.3 

Looking  at  the  rankings  by  expert  in  Table  4.3  only  makes  the  similarities  between  A,  B, 


and  C’s  rankings  more  obvious.  What  differences  there  are  will  be  discussed  below. 

4.2  Results  Grouped  by  Technical  Risk  Estimate 

4.2.1  First  Estimate.  Figure  4.1  shows  the  different  utilities  generated  by  the  three  sets  of 
preferences  for  the  technology  estimates  provided  by  the  first  expert.  The  technologies  are  shown 
in  order  of  B’s  rankings,  from  #1  at  the  bottom  to  #10  at  the  top  of  the  graph.  The  length  of  the 
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bars  show  the  large  differences  in  magnitude  between  the  different  “decision  maker’s”  utilities. 
The  technologies  are  ranked  based  on  each  “decision  maker’s”  utility  individually,  however,  so 
only  the  relative  differences  within  A,  B,  or  C’s  preferences  are  relevant.  The  relative  positioning 
of  the  technologies  are  almost  identical  despite  their  different  utility  surfaces. 

B  and  C  differ  in  ranking  AC  Arc  Melters  (50%,  50%)  vs.  the  Russian  Hi-Frequency 
Induction  method  and  In  Situ  Vitrification  (75%,  25%)  —  C  gives  more  value  to  the  high  P(dev) 
-  low  P(use)  combination  than  B  does.  The  difference  between  B  and  C  on  the  Plasma  Hearth 
Process  and  the  Plasma  Centrifical  Furnace  also  illustrates  something  interesting.  C  gives  them 
nearly  the  same  utility  of  .019  (rounded  off),  despite  having  different  attributes.  These  two 
technologies  lie  on  an  indifference  curve  for  C,  but  not  for  B. 


Total  Scores  by  Technology 

using  1st  estimate  &  all  3  preferences 


Figure  4. 1 
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Figures  4.2  through  4.4  show  the  indifference  curves  of  A,  B,  and  C  combined  with  the 
estimated  likelihoods  of  successful  development  and  use  provided  by  the  first  expert.  Again,  the 
colored  lines  represent  the  utility  contours  at  0.1  intervals,  starting  with  the  yellow  at  the  lowest 
left  and  climbing  hight  as  one  moves  to  the  upper  right.  A’s  indifference  curves  clearly  show  how 
only  T1  and  T2  are  viable  candidates  in  his  judgement.  The  difference  in  rank  between  T3  and  the 
T4/T5  pair  can  also  be  easily  seen  in  B’s  and  C’s  graphs. 

Figures  4.5  -  4.7  are  the  succinct  culmination  of  the  utility  assessment  process.  The 
choices  between  the  different  alternatives  are  clearly  visible.  The  shape  of  the  utility  the  trade-offs 
the  decision  maker  is  willing  to  make  between  the  two  technical  risk  attributes  to  get  the  best¬ 
valued  alternatives.  The  different  technologies  can  be  separated  into  different  groupings,  like 
“acceptable/unacceptable”  or  “high  risk/medium  risk/low  risk,”  as  desired  by  drawing  contour 
hues.  The  differences  between  the  three  “decision  makers’”  values  are  also  easily  seen  from  A’s 
flat,  discontinuous  shape  to  B’s  concave  surface  to  C’s  convex  one.  The  sensitivity  of  the 
rankings  to  changes  in  one  or  both  attributes  is  easy  to  see  from  the  color  contours.  For  example, 
T3  would  advance  in  rank  in  front  of  T4  and  T5  with  respect  to  C’s  preferences  if  its  estimated 
likelihood  of  success  improved  by  only  10%  or  so,  while  T7  would  require  a  vast  improvement  in 
P(dev),  P(use),  or  both  before  C  would  prefer  it  to  T4  and  T5. 
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A's  Surface,  Estimate  1 


Figure  4.5 
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utility 


B's  Surface.  Estimate  1 


Figure  4.6 
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Figure  4.7 
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4.2.2  Second  Estimate.  The  second  expert  had  rather  different  views  of  the  ten 
technologies’  technical  risk  factors  than  the  first.  As  one  can  see  from  Table  3.1,  many  of  the 
estimated  attributes  changed.  Consequently  the  ranking  of  the  technologies  has  changed.  The 
only  disagreement  between  B  and  C’s  rankings,  however,  is  that  C  thought  fairly  highly  of  Vortec 
Combustion,  coded  T9,  compared  to  B’s  #5-#7  technologies  (T8,  T7,  &  T6,  respectively).  A 
look  at  the  contour  plots  below  reveals  that  the  difference  results  from  C  placing  practically  no 
value  on  any  technology  with  a  P(dev)  of  less  than  48%  and  a  P(use)  of  less  than  roughly  60%. 
The  yellow-colored  indifference  curve  corresponds  to  a  combined  utility  of  .1,  which  is  consistent 
with  the  utilities  shown  in  Figure  4.8. 

Total  Scores  by  Technology 

using  2nd  estimate  &  all  3  preferences 


Figure  4.8 
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P(use) 


B’s  Surface  (contours).  Eslimate  2 


The  tight  cluster  of  Tl,  T2,  and  T4  near  the  upper  corner  of  tlie  utility  surfaces  results  in 
nearly  equivalent  utilities.  Other  criteria  arc  required  should  it  be  necessary  for  these  “decision 
makers”  to  distinguish  between  them.  B’s  relative  tolerance  for  treatment  technologies  expected 
to  be  only  moderately  successful  is  shown  here  by  the  scattering  of  mid-range  technologies  (T3, 
T8,  T7,  T6,  T9,  &  T5)  across  his  concave  utility  surface.  The  convexity  of  C’s  surface  reflects 
his  contrasting  intolerance  for  mediocre  technologies.  His  preference  for  T9  over  T8,  T7,  and  T6 
is  easy  to  see,  while  B’s  preference  for  T8  and  T7  is  equally  as  evident. 


A's  Surface,  Estimate  2 


Figure  4.12 
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C's  Surface,  Estimate  2 


Figure  4.14 


4.2.3  Third  Estimate..  The  third  expert  was  generally  more  conscryativc  in  his  estimates 
of  P(dev)  and  P(use),  with  cones pondingly  lower  utilities  across  the  board.  Figure  3.3  shows 
how  the  estimates  strung  the  technologies  in  a  rough  line  between  (0,0)  and  (1,1).  One  would 
expect  rankings  to  correspond  to  the  technology’s  position  in  this  “string,”  and  they  do. 


T1 

T2 

T3 

T4 

T5 

T6 

T7 

T8 

T9 

TIO 

Joule 

DC 

AC  Arc 

In  Situ 

Russian 

Russian 

Plasma 

Plasma 

Vortec 

Micro¬ 

Heated 

Graphite 

Mclters 

Vitrifi¬ 

Hi-Freq. 

Hybrid 

Hearth 

Cent. 

Combust. 

wave 

Mclters 

Plasma 

cation 

Induction 

Plasma 

Process 

Furnace 

Mclters 

4-15 


Total  Scores  by  Technology 

using  3rd  estimate  &  all  3  preferences 


Figure  4.16 


Figure  4.17 
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Figure  4.15 


Figure  4.18 
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A’s  Surface,  Estimate  3 


Figure  4.19 


This  expert’s  estimates  are  not  so  interesting.  Since  every  technology  in  the  “string”  has  a 
better  P(dev)  and  P(use)  than  the  one  immediately  following,  the  choices  are  clear.  This  situation, 
when  one  alternative  is  clearly  better  than  others,  is  called  dominance  [Clemen,1991:88, 437]. 
Here  we  can  say  that  T1  dominates  all  the  other  candidate  technologies,  T4  dominates  the 
remaining  ones,  and  so  on  for  this  estimate.  Dominance  greatly  simplifies  decisions  when  it 
occurs. 
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C‘s  Surface.  Estimate  3 


Figure  4.21 


4.2.4  Fourth  Estimate.  The  fourth  expert  was  more  optimistic  with  his  estimates  than  the 
third  was,  but  his  estimates  also  form  a  kind  of  “string”  (see  Figure  4.4).  This  “string”  curves  out 
in  a  rough  parabola,  with  every  technology  dominating  its  predecessor.  A,  B,  and  C  all  agree  on 
the  rankings  of  these  technologies.  There  are  several  technologies  with  the  same  risk  estimates, 
making  other  criteria  the  distinguishing  factors. 
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using  4th 


Figure  4.23 
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Figure  4.24 
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Figure  4.25 
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C's  Surface,  Estimate  4 
T1:J4 


Figure  4.28 


4.3  Sensitivity  Analysis 

The  results  summarized  in  Table  4.2  are  very  much  the  same  for  all  three  “decision 
makers”  and  experts’  estimates,  despite  the  different  utility  surfaces  of  A,  B,  and  C.  One  reason 
for  this  similarity  of  rankings  is  that  two  of  the  technology  attribute  estimates  have  almost  every 
technology  dominated  by  its  immediate  predecessor.  Any  utility  surface  would  generate  the  same 
rankings  for  the  third  and  fourth  estimates. 
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4.3. 1  Russian  Hi-Frequency  Induction  Method.  The  agreement  of  the  rankings  between 
“decision  makers”  and  expert  estimates  is  striking  when  Table  4.2  is  studied.  One  apparent 
inconsistency,  however,  is  the  difference  in  rank  of  the  Russian  Hi-Frequency  Induction  method, 
T5,  between  estimates  1  &  4  and  estimates  2  &  3.  In  estimate  1,  T5  is  ranked  4*  (by  B)  or  3"^^  (by 
C).  In  estimate  4,  it  is  3'**  for  all  three  “decision  makers.”  However,  T5  is  ranked  9th  in  estimate 
2  and  8*  (in  a  three  way  tie  for  last)  in  estimate  3  by  B  and  C. 

The  reason  for  this  anomaly  can  be  seen  in  Table  3.1.  The  P(dev),  P(use)  numbers  are 
(75, 25)  in  the  first  estimate,  (80, 100)  in  the  fourth,  (20, 30)  in  the  second,  and  (10, 10)  in  the 
third.  This  is  the  widest  disagreement  between  the  four  experts  and  is  worth  further  analysis. 

If  one  examines  the  other  candidate  technologies  ranked  about  T5,  one  can  see  the 
changes  in  P(dev)  and  P(use)  that  would  be  necessary  to  raise  or  lower  T5's  ranking  as  a  measure 
of  how  sensitive  T5's  rank  is  to  the  estimated  risk  factors.  Table  4.4  lists  what  it  would  take  in 
positive  and  negative  adjustments  to  each  expert’s  P(dev)  and  P(use)  to  change  the  rank  of  T5. 

Table  4.4  requires  some  explanation.  It  should  be  read  down  each  column,  one  for  each 
expert’s  estimates,  while  looking  at  the  rows  associated  with  one  “decision  maker’s”  preferences 
at  a  time.  Only  one  attribute  was  allowed  to  change  at  a  time.  The  rows  show  what  change  to 
T5's  P(dev)  or  P(use)  would  be  necessary  to  force  a  change  in  rank.  The  changes  in  the  attributes 
raise  or  lower  T5's  utility  to  the  same  value  as  the  next  highest  or  lowest  ranked  technology, 
thereby  changing  T5's  rank. 
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One-Way  Sensitivity  Analysis  for  Russian  Hi-Frequency  Induction 

changes  in  P(dev)  or  P(use)  that  change  the  rank  of  the  technology 
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can’t 

lower 
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at  bottom 

at  bottom 

at  bottom 

0 

A  P(use) 

at  bottom 

at  bottom 

at  bottom 

0 

B’s  preferences 

raise 

A  P(dev) 

can’t 

+  21% 

+  16% 

+20% 

A  P(use) 

+  16% 

+  17% 

+  07% 

can’t 
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A  P(dev) 

0 

can’t 
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0 

A  P(use) 

0 

-20% 

at  bottom 

0 

C’s  preferences 

raise 

A  P(dev) 

can’t 

+  12% 

+  06% 

+  20% 

A  P(use) 

+  50% 

+  20% 

+  31% 

can’t 

lower 

A  P(dev) 

0 

-01% 

1 

at  bottom 

0 

A  P(use) 

0 

-20% 

at  bottom 

0 

Table  4.4 


Since  so  many  technologies  lie  below  A’s  75%  thresholds,  in  many  cases  T5  has  the  same 
zero  utility  as  a  technology  with  a  higher  P(dev)  or  P(use).  In  these  cases  it  is  meaningless  to 
consider  raising  T5's  rank,  signified  by  the  entry  “next  below  threshold.”  “At  bottom”  refers  to 
situations  where  T5  is  already  at  the  bottom  of  the  technology  rankings  and  therefore  can  not 
lower  in  rank.  It  is  possible  that  no  improvement  in  one  attribute  can  improve  the  rank  of  T5 
because  even  a  100%  attribute  would  not  improve  T5’s  utility  enough  to  reach  the  next  highest 
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technology.  Conversely  sometimes  the  next  lowest  technology’s  utility  is  too  low  to  be  reached 
by  changing  only  one  attribute.  A  “can’t”  entry  refers  to  these  cases.  Finally,  if  T5  has  the  same 
attributes  as  another  technology,  even  an  infinitesimal  change  in  an  attribute  would  change  the 
rank  of  T5.  This  is  signified  by  a  “0.” 

For  example,  if  one  was  examining  the  sensitivity  of  T5's  rank  using  B’s  preferences  in 
estimate  2,  one  would  see  a  rise  in  rank  by  increasing  P(dev)  by  21%  (to  41%)  or  by  increasing 
P(use)  by  17%  (to  47%).  One  couldn’t  lower  T5's  rank  by  dropping  P(dev),  but  it  is  possible  to 
drop  T.5  to  last  place  by  decreasing  P(use)  by  20%  (to  10%). 

The  most  sensitive  cases  are  those  where  the  Russian  Hi-Frequency  Induction  method 
shares  the  same  P(dev)  and  P(use)  estimates  with  other  treatment  technologies.  Readdressing 
these  estimates  with  additional  care  would  be  advisable,  since  any  change  will  cause  a  shift  in 
rankings.  Other  cases  where  the  necessary  changes  are  less  than  10%  should  be  treated  with 
similar  care.  The  most  interesting  case  is  with  the  first  expert’s  estimates,  where  T5  is  in  the  top 
four  technologies  and  only  an  increase  of  16%  would  be  needed  in  P(use)  to  shift  B’s  rankings. 
The  situation  in  estimate  4  is  not  as  interesting,  since  T5's  P(dev)  would  have  to  rise  to  100%  to 
increase  in  rank,  joining  T1  and  T4  at  complete  certainty  (100%,  100%). 

4.3.2  Vortec  Combustion  in  Estimate  Two.  One  other  anomaly  worth  investigating  is  the 
difference  in  Vortec  Combustion’s  ranking  (T9)  using  the  second  expert’s  estimates  of  (70%, 
20%)  and  B  and  C’s  preferences.  B  rates  T9  as  #8,  while  C  gives  it  #5.  That  is  a  significant 
difference  when  using  the  same  attributes. 
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B’s  utility  function  gives  T9  a  0.585  utility,  which  is  fairly  low  for  B.  C  gives  T9  a  0.166 
utility,  enough  to  distinguish  it  from  T8,  T7,  and  T6  as  mentioned  in  section  4.2.2.  An 
examination  of  Figures  4.10  and  4.1 1  or  4.13  and  4.14  show  how  C’s  relatively  lower  value  for 
low  P(dev)  estimates  lowers  the  other  technologies’  ranks  relative  to  B’s  rankings.  B’s 
indifference  curves  in  Figure  4. 10  show  the  blue  .6  utility  contour  passes  just  over  T9's 
coordinates  but  beneath  T6,  T7,  and  T8. 

Since  the  indifference  curves  seem  to  suggest  the  difference  is  caused  by  C’s  higher  value 
for  P(dev)  than  B,  a  one  way  sensitivity  analysis  on  the  weights  of  the  multi-attribute  utility 
functions  seems  in  order.  Again,  the  idea  is  to  find  the  point  where  two  technologies  have  the 
same  total  utility.  Here  T9  will  be  checked  against  T6  and  T7/r8  to  find  levels  of  and  where 
the  utilities  are  equal.  The  results  are  summarized  in  Table  4.5  below. 

C  has  an  additive  multi-attribute  utility  function  (see  equation  1  in  section  1. 2.4.1).  This 
form  requires  k^  +  k„  =  1.  Therefore  we  only  need  to  examine  the  one  case.  Since  B’s  multi¬ 
attribute  utility  function  is  has  a  multilinear  form  (see  equation  2  in  section  1. 2.4.2),  k^,  and  ku  do 
not  have  to  sum  to  1,  so  they  have  to  be  analyzed  separately  while  the  other  stays  constant. 
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Sensitivity  Analysis  for  Vortec  Combustion  in  Estimate  2 

changes  in  weights  that  force  T9  to  change  ranks  with  other  technologies 


C’s  utility  function  {k^=l  -  k^ 

current  value 

weights  needed  to  lower  T9's  rank 

kd 

0.25 

0.01 

K 

0.75 

0.99 

resulting  utility  of  T9 

0.166 

0.007 

B’s  utility  function  *  1  -  kj) 

current  value 

weights  needed  to  raise  T9's  rank 

kd 

0.3863 

0.457 

holding  constant 

0.7726 

0.7726 

^-kd-K 

-0.1589 

-0.2296 

resulting  utility  of  T9 

0.601 

0.64 

ku 

0.7726 

0.7125 

holding  kj  constant 

0.3863 

0.3863 

1  - 

-0.1589 

-0.0988 

resulting  utility  of  T9 

0.601 

0.599 

Table  4.5 


The  results  in  Table  4.4  need  some  interpretation.  To  drop  Vortec  Combustion  in  C’s 
rankings,  practically  no  importance  could  be  placed  on  the  risks  of  successful  development.  The 
only  thing  that  could  matter  would  be  the  estimated  success  of  the  technique  in  the  field.  This  is 
unrealistic  and  can  be  dismissed  as  a  possibility.  Therefore  C  would  never  really  change  his 
ranking  of  Vortec  Combustion  if  the  second  expert’s  P(dev)  and  P(use)  estimates  were  used. 
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The  results  for  B  are  less  clear.  Since  his  combined  utility  function  is  not  additive,  it  is 
more  difficult  to  interpret  what  the  weights  mean.  When  changing  to  cause  T9  to  switch  ranks 
with  T6,  more  importance  is  placed  on  P(dev)  than  before.  Less  importance  is  placed  on  P(use) 
when  changing  k^.  While  this  is  consistent,  the  full  picture  must  include  the  (1  -  k^  -  k^)  term. 
Using  the  concepts  of  complements  and  substitutes  introduced  in  section  1.2.4.2.1,  one  can  see 
that  B  treats  P(dev)  and  P(use)  as  substitutes  —  a  shortfall  in  one  attribute  can  be  made  up  for  by 
a  high  estimate  in  the  other.  This  relationship  becomes  more  pronounced  when  k^  is  raised, 
making  the  attributes  even  more  able  to  act  as  replacements  for  each  other.  When  k^  is  lowered, 
however,  the  1  -  kj  -  k^  becomes  almost  insignificant,  suggesting  that  B’s  ranking  of  the  treatment 
technology  is  sensitive  to  his  original  dismissal  of  additive  independence. 
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V.  Conclusions 


While  the  risks  involved  with  procuring  emerging  technologies  are  large,  new  remediation 
techniques  must  still  be  developed  in  order  to  affordably  achieve  the  long-term  remediation 
objectives  of  the  Department  of  Energy.  Minimizing  the  technical  risks,  as  well  as  the  financial 
and  environmental  costs  they  induce,  is  one  way  the  Office  of  Technology  Development  is 
meeting  these  objectives.  This  requires  intelligent  decision  making  about  the  candidate 
technologies,  since  these  decisions  will  not  be  simple.  The  models  developed  in  this  study  show 
how  one  set  of  risk  trade-offs  can  be  made  with  the  assistance  of  decision  analysis  techniques. 

Two  important  aspects  of  technical  risk  were  examined  from  a  DOE  manager’s  viewpoint. 
The  risk  that  a  technology  being  developed  would  not  be  ready  for  use  in  the  field  requires  that  a 
manager  face  the  possibility  of  having  expended  critical  time  and  funds  on  a  technique  that  cannot 
fulfill  scheduled  remediation  deadlines.  The  risks  of  failure  in  use  carry  graver  consequences,  with 
possibilities  of  wasted  effort  and  violated  regulatory  standards  or  even  of  inadvertent  exposure  of 
radioactive  and  hazardous  materials  to  people  and  the  environment.  Other  concerns,  such  as 
development  cost  and  schedule  risks,  were  not  addressed  in  this  study. 

5.1  The  Assumptions 

The  structure  of  the  modeled  preferences  in  this  study  depends  on  the  definitions  of  the 
two  technical  risk  measures,  P(dev)  and  P(use).  Based  upon  the  definitions  of  P(dev)  and  P(use), 
and  the  assumptions  within  them,  the  utility  functions  developed  in  this  study  are  useful  to  map 
the  trade-offs  between  development  and  implementation  risk.  The  rankings  of  the  ten  treatment 
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technologies  were  much  more  sensitive  to  the  various  expert  estimates  than  to  these  assumptions, 
and  so  further  refinement  and  sophistication  was  deemed  unnecessary  considering  the  coarseness 
of  the  data. 

5.1.1  Development  Risk.  The  risk  of  successful  development  was  defined  as  likelihood 
that  a  technology  would  have  completed  research  and  development  by  a  certain  date  and  be  ready 
for  employment  in  the  field.  This  probability  is  defined  to  be  independent  of  the  time  spent  in 
R&D,  through  the  assumption  that  the  performance  standards  the  technology  must  pass  in  the 
demonstration  and  validation  stage  remain  the  same  throughout  R&D.  The  estimates  of  this 
likelihood  do  not  take  any  budgetary  effects  into  account  —  increasing  or  decreasing  the  funding 
to  a  development  project  can  accelerate  or  retard  the  progress  toward  a  commercializable 
product,  effectively  raising  or  lowering  estimates  of  P(dev).  However,  the  difficulty  of  predicting 
from  present  day  knowledge  future  overall  funding  fluctuations  or  the  effects  of  prioritizing 
certain  projects  over  others  practically  precludes  the  inclusion  of  such  items  into  this  study. 

5.1.2  Implementation  Risk.  The  risk  of  successful  use  is  carefully  separated  by  its 
definition  from  the  duration  of  a  technology’s  R&D  period.  This  is  to  exclude  unknowable 
factors  from  this  study,  since  the  dependence  of  a  technology’s  future  performance  on  R&D  that 
has  not  been  completed  or,  in  some  cases,  even  begun,  is  difficult  to  estimate.  Thus  the  definition 
of  the  probability  of  successful  field  use  is  the  likelihood  of  a  technology  that  meets  its 
demonstration  and  validation  standards  accomplishing  its  designed  purpose.  Note  that  the  nature 
of  failure  in  actual  use  is  left  unspecified  —  while  some  failures  may  be  relatively  minor,  requiring 
perhaps  that  another  technique  must  be  used  to  finish  a  job  left  uncompleted,  there  is  the  potential 
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for  catastrophic  accidents  involving  inadvertent  exposure  to  toxic  or  radioactive  waste.  The 
prediction  of  possible  consequences  of  an  unknown  failure  are  beyond  the  scope  of  this  study. 

The  nature  of  the  landfill  itself  is  left  out  of  these  P(use)  estimates.  The  estimates  of  P(use)  must 
be  conditional  on  the  assumption  that  these  preliminary  site  assessments  are  correct,  since  the 
potential  effects  of  inaccurate  preliminary  site  characterization  are  unknowable. 

5.1.3  Subjective  Probability  Estimates.  Since  this  study  focused  on  modeling  decision 
makers’  attitudes  toward  technical  risk,  very  little  attention  was  paid  to  the  technology  attributes 
estimated  from  expert  opinion.  The  subject  of  subjective  probability  estimation  is  too  broad  to  be 
summarized  here,  but  one  thing  is  clear.  Technology  forecasting  of  innovative  items  has  no  other 
choice  but  to  use  expert  opinion  as  the  source  of  needed  information  simply  because  no  objective 
data  exists  for  emerging  technologies.  Expert  estimates  are  often  biased  and  inaccurate  [see 
Hudak,  1994;  King  &  Wilson,  1967;  King,  et.  al.,  1967;  Biery,  et.  al.,  1994].  Errors  are  even 
more  likely  when  dealing  with  innovative  technology,  since  the  developers  of  the  technology 
themselves  may  be  the  only  one  possessing  the  necessary  understanding  to  give  credible  opinions, 
and  they  most  often  have  optimistic  views  on  the  matter  to  justify  their  work  to  themselves  and 
their  sponsors.  Despite  the  inaccuracy  of  expert  opinion  and  subjective  probability  estimates, 
there  is  no  other  alternative  to  using  it. 

The  difference  in  rankings  of  the  ten  treatment  technologies  in  section  4  of  this  report 
result  more  from  the  differences  in  experts’  estimates  of  P(use)  and  P(dev)  than  from  the 
preference  stmetures  of  the  “decision  makers.”  The  implications  of  this  are  clear:  when  applying 
the  methods  discussed  in  this  study  to  a  real  decision,  at  least  as  much  attention  must  be  placed 
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on  getting  quality  estimated  risk  attributes  as  on  correctly  modeling  the  decision  maker’s 
preferences.  The  old  computer  programming  rule,  “garbage  in,  garbage  out”  definitely  applies. 

5.2  The  Results 

Based  on  the  limited  sample  utilized  in  this  study,  the  best  choices  for  low  technical  risk 
treatment  technologies  remained  Joule  Heated  Melters,  In  Situ  Vitrification,  DC  Graphite  Plasma 
Arcs,  and  AC  Arc  Melters  (Tl,  T4,  T2,  and  T3,  respectively),  across  the  majority  of  combinations 
of  experts’  estimates  and  “decision  makers’”  utility  surfaces.  These  technologies  have  the  least 
anticipated  technical  risk  for  treatment  processes  scheduled  to  begin  seven  years  from  now. 

Despite  the  different  viewpoints  of  the  three  interviewed  “decision  makers”  and  the 
resulting  mathematical  descriptions  of  their  values  on  technical  risk,  the  rankings  of  candidate 
treatment  technologies  remained  consistently  similar.  Each  of  the  three  contractor’s  utility 
surfaces  were  assessed  independently.  This  suggests  that  using  multi-attribute  utility  theory  to 
model  tills  aspect  of  a  technology  selection  decision  has  at  least  “face  validity,”  and  that  one  could 
reasonably  apply  these  methods  to  other  similar  decision  problems. 

The  Russian  Hi-Frequency  Induction  technology  was  the  only  candidate  whose  ranking 
greatly  varied  between  expert  estimates  (see  section  4.3.1).  The  estimates  for  this  technology 
showed  the  only  serious  inconsistency  between  experts.  Since  this  technology  was  ranked  in  the 
top  four  for  two  of  the  experts’  estimates,  this  particular  technology  should  be  examined  with  care 
to  analyze  its  technical  risks  before  final  selection  of  technologies  is  made.  This  is  the  issue  where 
additional  study  will  have  the  greatest  impact  on  the  decision. 
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The  three-dimensional  graphs  in  section  5  of  this  report  were  found  to  condense  many  of 
the  issues  of  risk  tolerance  and  technology  comparison  into  a  concise  form  that  is  easy  to 
understand.  Dlustrations  of  this  sort  can  summarize  results  that  would  otherwise  take  several 
pages  of  tables  and  easily  misunderstood  explanation  to  present.  The  sensitivity  of  the  rankings  of 
alternatives  to  changes  in  their  attributes  are  obvious  and  easily  explained. 

5.3  How  to  Use  This  Study 

5.3.1  Selecting  a  Treatment  Technology.  By  examining  the  utility  surface  plots  for  each 
set  of  estimates  in  section  4,  one  can  see  how  close  the  lower  ranked  technologies  come  to  the 
best  ones  in  order  to  anticipate  how  other  non-risk  factors  might  change  a  decision  maker’s  order 
of  preference.  For  example,  when  considering  the  second  expert’s  estimates  as  depicted  on 
Figures  5.12, 5.13,  and  5.14,  one  can  easily  see  that  A  would  never  consider  T3  as  a  potential 
candidate,  while  B  would  see  T3  as  only  somewhat  less  preferable  than  T2,  T4,  or  Tl.  C  views 
T3  as  more  suitable  than  lesser  ranked  technologies,  but  other  considerations  would  have  to  carry 
very  high  weight  before  he  would  prefer  it  in  place  of  the  three  substantially  less  risky  alternatives. 

5.3.2  Models  of  "Decision  Maker”  Preferences.  Just  as  the  various  estimates  of  the 
development  and  implementation  risks  for  the  ten  treatment  technologies  shown  in  Figure  1.1 
were  applied  to  the  three  “decision  makers’”  utility  surfaces,  any  set  of  technology  alternatives 
can  be  ranked  by  use  of  the  appropriate  process’  utility  surfaces.  The  three  contractors  whose 
preferences  were  assessed  had  very  different  viewpoints  (programmatic,  waste  site  manager,  and 
technology  developer)  and  their  utility  surfaces  could  be  applied  as  rough  characterizations  of 
decision  makers  who  share  those  perspectives. 
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It  should  be  noted  that  these  utility  functions  are  independent  of  any  specific  technology 
alternatives  —  unlike  pairwise  comparison  methods  like  the  Analytic  Hierarchy  Process  (AHP), 
the  utility  functions  described  in  section  2  are  valid  for  all  sets  of  potential  candidates  one  might 
want  to  examine  with  regard  to  these  attributes  of  technical  risk,  and  do  not  have  to  be  re¬ 
assessed  whenever  new  candidates  are  added  the  way  AHP  results  must  be  [Kloeber,  1992:15]. 

5.3.3  Methodology  Applied  to  Other  Decisions.  This  approach  mapped  the  decision 
maker’s  value  to  quantified  risk  attributes  of  potential  candidate  technologies.  Other  attributes, 
such  as  cost  or  schedule  estimates,  could  easily  be  examined  in  the  same  manner.  All  that  is 
required  is  careful  definition  of  the  attributes,  then  an  interview  with  the  decision  maker  where  his 
or  her  utility  functions  are  assessed.  Some  follow-up  work  by  the  interviewing  analyst  is 
necessary  afterwards  to  produce  graphical  summaries  of  the  decision  maker’s  utility  functions  and 
analysis  of  the  resulting  ranking  of  alternatives.  In  general,  with  proper  background  and  software 
support,  this  is  not  an  extremely  difficult  process  to  complete  for  relatively  simple  trade-offs,  like 
that  between  developmental  and  implementation  risk  here. 

One  thing  to  emphasize,  however,  is  that  this  kind  of  study  is  the  result  of  a  partnership 
between  the  decision  maker  and  the  analyst.  In  a  sense,  the  analyst  helps  the  decision  maker 
quantitatively  express  his  or  her  values  for  certain  parts  of  the  decision  in  order  to  make  better 
overall  choices.  Utility  curves  are  no  replacement  for  the  decision  maker  whose  preferences  are 
mapped  —  the  decision  maker  “owns”  the  decision,  while  the  analyst  is  providing  a  service  whose 
ultimate  quality  is  measured  by  satisfaction  of  the  decision  maker.  Analysis  of  the  sort  described 
in  this  report  can  only  illuminate  part  of  the  decision  problem  through  quantitative  means  such  as 
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utility  theoiy.  Other  qualitative  factors  relevant  to  the  decision,  such  as  political  or  budgetary 
constraints,  are  left  to  the  decision  maker  to  sort  out,  since  he  or  she  is  the  best  judge  of  their 
importance  and  impact. 

While  the  graphs  of  the  various  utility  curves  and  the  candidate  technologies  are  an 
important  outcome  of  this  work,  the  primary  result  has  been  the  demonstration  of  multi-attribute 
utility  theory  to  formalize  a  decision  maker’s  feelings  about  technical  risk,  for  use  in  remediation 
technology  selections.  The  three  interviews  took  about  five  hours  total  to  accomplish,  and  yet 
provided  enough  information  to  develop  a  powerful  model  for  analyzing  investment  decisions 
about  emerging  technologies.  The  encapsulation  of  the  experience  and  preferences  of  upper-level 
managers  is  not  perfect  by  any  means,  but  a  model  like  the  ones  developed  in  this  project  can  help 
decision  makers  focus  on  the  equally  important  qualitative  issues.  Together  with  other  decision 
analysis  tools,  preference  models  based  on  utility  theory  can  be  applied  to  assist  technology 
managers  in  making  the  tough  choices  necessary  to  carry  out  the  Department  of  Energy’s  long¬ 
term  envhonmental  remediation  program. 
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Appendix  A  —  "Decision  Makers”  and  Experts 


The  assistance  of  all  involved  with  this  project  was  greatly  appreciated.  The  “decision 
makers”  whose  utilities  were  assessed  were  George  Dewhirst,  Paul  Krumrine,  and  Dave  Simpson, 
working  at  the  Waste  Policy  Institute’s  field  office  at  Aiken,  South  Carolina.  Tlie  experts  who 
provided  their  opinions  about  the  ten  treatment  technologies  were  Drs.  Leo  Baetsle,  Erwin 
Fenjrves,  William  Prindle,  and  Henry  Schreiger. 
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Appendix  B  —  Equations  of  Utility  Curves 

The  utility  curves  and  surfaces  shown  in  section  III  of  this  report  were  generated  with  the 
help  of  the  Logical  Decisions  for  Windows®  program  (LDW).  After  the  single-attribute  utility 
functions  were  assessed  during  the  interview,  they  were  transferred  to  LDW®  by  replicating  the 
hand-drawn  curves  using  the  “Assess  SUF”  procedure.  The  weights  for  the  multi-attribute  utility 
functions  were  assessed  using  the  on-screen  procedures  of  LDW®.  The  weights  and  utility 
equations  were  then  re-written  into  a  spreadsheet  to  generate  the  data  plotted  in  the  report. 

LDW®  fitted  the  utility  curves  with  either  a  linear  or  an  exponential  equation  applied  piece- 
wise  across  the  range  of  the  attributes.  The  exponential  equation  had  this  form: 

y  =  a  +  b  e'^^  (A-1) 

where  x  is  the  attribute  in  question.  The  linear  equation  had  this  form: 

y  =  a  +  b  X 

These  equations  and  the  multi-attribute  weights  follow,  for  each  “decision  maker’s”  utility 
function  shown  in  the  report.  If  c  =  0  in  the  following  list,  use  the  linear  form. 


A 


all;  k,  =  0.2315,  k„  =  0.7685 

P(dev)  &  P(use)  0 

-1 

B 

characterization;  k^  =  0.4137,  k^  =  0.8273 
P(dev)  0.04003 

-0.05 
0.08571 
1 

P(use)  -0.0158 

1.011 

stabilization;  k^  =  0.2269,  k^  =  0.4537 
P(dev)  0.04003 

-0.05 
0.08571 
1 


0 

0 

(0, 0.75) 

2 

0 

(0.75, 1) 

-0.04003 

-36.64 

(0,0.2) 

0.04555 

3.406 

(0.2, 0.5) 

0.001998 

8.093 

(0.5, 0.75) 

6.924E-I-08 

-31.13 

(0.75,  1) 

0.0158 

7.68 

(0. 

-7.68 

-6.563 

(0.5, 1) 

-0.04003 

-36.64 

(0,0.2) 

0.04555 

3.406 

(0.2,  0.5) 

0.001998 

8.093 

(0.5, 0.75) 

6.924E+08 

-31.13 

(0.75,  1) 

B-1 


“Decision  Maker” 

technology  type  &  multi-attribute  weights  Range  of  Equation 


attribute 

a 

b 

c 

(from,  to) 

P(use) 

-0.01286 

0.01286 

7.525 

(0. 0.32) 

-0.5942 

0.4663 

1.376  (0.32,0.62) 

1.018 

-127.2 

-8.78 

(0.62,  1) 

containment;  k^  =  0.3197,  k„  =  0.6394 

P(dev) 

0.04003 

-0.04003 

-36.64 

(0,0.2) 

-0.05 

0.04555 

3.406 

(0.2, 0.5) 

0.08571 

0.001998 

8.093 

(0.5, 0.75) 

1 

6.924E+08 

-31.13 

(0.75,  1) 

P(use) 

0 

0.05 

0 

(0,0.1) 

-0.3571 

0.292 

2.154 

(0.1, 0.5) 

1.146 

-2.856 

-2.972 

(0.5, 1) 

removal;  k^  =  0.1771,  k„  =  0.3543 

P(dev)  0.04003 

-0.04003 

-36.64 

(0, 0.2) 

-0.05 

0.04555 

3.406 

(0.2, 0.5) 

0.08571 

0.001998 

8.093 

(0.5, 0.75) 

1 

6.924E+08 

-31.13 

(0.75,  1) 

P(use)  -0.03242 

0.03242 

4.665 

(0, 0.4222) 

-0.1245 

0.7685 

0 

(0.4222,  0.62) 

0.3198 

3.963E-08  21.95 

(0.62, 0.73) 

1.061 

-53.76 

-6.78 

(0.73, 1) 

treatment  &  disposal;  k^  =  0.3863,  Iq,  = 
P(dev)  -0.09497 

0.7726 

0.09497 

5.821 

(0,0.2) 

1.341 

-1.745 

-2.166 

(0.2, 0.5) 

1.047 

1.876 

-3.687 

(0.5, 1) 

P(use)  1.125 

-1.125 

-2.197 

(0, 1) 

monitoring;  =  0.0334,  k„  =  0.0667 


B-2 


“Decision  Maker” 

technology  type  &  multi-attribute  weights  Range  of  Equation 


attribute 

a 

b 

c 

(from,  to) 

P(dev) 

0 

1 

0 

(0, 1) 

P(use) 

-0.3346 

0.3346 

1.203 

(0, 0.76) 

2.856 

-8.829 

-1.738 

(0.76, 0.82) 

1.019 

-6.553E+04 

-15.05 

(0.82, 1) 

C 

characterization,  stabilization,  containment,  &  monitoring;  =  0.25,  ky  = 

0.75 

P(dev) 

-0.0035 

0.0035 

13.86 

(0, 0.2) 

-0.0796 

0.07007 

3.175 

(0.2, 0.5) 

1.329 

-3.453 

-2.351 

(0.5, 1) 

P(use) 

-0.025 

0.025 

4.394 

(0, 0.5) 

1.1 

-8.1 

-4.394 

(0.5, 1) 

removal,  treatment,  &  disposal;  k,,  =  0.25,  Iq, 

=  0.75 

P(dev) 

-0.0035 

0.0035 

13.86 

(0,0.2) 

-0.0796 

0.07007 

3.175 

(0.2, 0.5) 

1.329 

-3.453 

-2.351 

(0.5, 1) 

P(use) 

-l.OE+05 

l.OE+05 

15.32 

(0,0.6) 

-1.1 

2 

0 

(0.6, 0.8) 

1.113 

-538.2 

-8.473 

(0.8, 1) 

B-3 
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